Method and system for anonymous measurement of online advertisement using offline sales

ABSTRACT

Anonymous measurement of online advertising using offline sales is disclosed. The measurement uses anonymized online ad exposure data and anonymized offline sales, or conversion data, to measure effectiveness of online advertisement. The anonymized exposure and conversion data are matched using tokens from each data set; each token corresponds to a user and is generated from personally identifiable information of the user that is anonymized to preserve the user&#39;s privacy. The anonymized exposure and conversion data is aggregated using the tokens to map records in the data sets for a given user. Users&#39; ad exposure and sales history is used, anonymously, to gauge ad effectiveness of a corresponding ad campaign. The effectiveness measure, or measures, can further be used to configure another ad campaign.

FIELD OF THE DISCLOSURE

The present disclosure relates to privacy protection in measurement ofonline advertising campaigns, and more particularly to measuringeffectiveness of online advertisement on offline sales in a privacyprotective manner.

BACKGROUND

A significant barrier to shifting advertisement budgets to the onlinechannel is the inability to measure the effectiveness and/or impact ofan online advertisement campaign on offline sales, e.g., brick andmortar sales. Measurement, or metrics, that have been used for onlinecampaigns are typically based on click through rate (CTR), surveys oroffline impact analysis based on a panel of users. Missing in themeasurement of online advertisement effectiveness is data from offlinesales. Since a significant percentage, e.g., approximately 95%, of thetotal commerce in the United States, is derived from offline sales, itwould be beneficial to be able to make use of data from offline sales todetermine the effectiveness of online advertising, and to measurecausality between exposure to an online display ad and an offline salesevent(s).

SUMMARY

The present disclosure addresses these and other needs. By way of anon-limiting introduction, the present disclosure comprises measurementof online ad effectiveness that uses anonymized online ad exposure dataand anonymized offline sales, or conversion, data to measureeffectiveness of online advertisement. The anonymized exposure andconversion data are matched using tokens from each data set; each tokencorresponds to a user and is generated from personally identifiableinformation of the user that is anonymized to preserve the user'sprivacy. The anonymized exposure and conversion data is aggregated usingthe tokens to map records in the data sets for a given user. Users' adexposure and sales history can be used, anonymously, to gauge adeffectiveness. The effectiveness measure, or measures, can be used toconfigure another ad campaign, including setting a budget for onlineadvertising.

In one aspect, a method is provided by at least one computing devicethat maintains a user data store for each of a plurality of users, eachuser data store storing personally identifiable information about acorresponding user, the corresponding user's personally identifiableinformation being associated with a unique identifier that uniquelyidentifies the user in ad tracking data; collects the ad tracking datafor each advertisement served to the plurality of users in connectionwith an online advertisement campaign; generates exposure data using thead tracking data, the exposure data including at least a portion of thepersonally identifiable information for each user of the plurality;causes a unique first token to be generated using a first identifierassociated with the personally identifiable information included in theexposure data for each user of the plurality, each unique first tokenbeing a first level of abstraction of the corresponding user'spersonally identifiable information included in the exposure data anduniquely identifying the corresponding user; causes a unique secondtoken to be generated using a second identifier associated with thepersonally identifiable information included in the exposure data foreach user of the plurality, each unique second token being a secondlevel of abstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; and causes each user's personally identifiableinformation in the exposure data to be replaced with the user'scorresponding unique second token, the unique second token used toidentify the corresponding user's exposure to the online ad campaign andto identify sales transactions of the corresponding user in offlinesales transaction data in measuring effectiveness of the onlineadvertisement campaign using the exposure and offline sales transactiondata.

In another aspect, a system is provided, the system comprises at leastone processor executing and memory storing instructions to maintain auser data store for each of a plurality of users, each user data storestoring personally identifiable information about a corresponding user,the corresponding user's personally identifiable information beingassociated with a unique identifier that uniquely identifies the user inad tracking data; collect the ad tracking data for each advertisementserved to the plurality of users in connection with an onlineadvertisement campaign; generate exposure data using the ad trackingdata, the exposure data including at least a portion of the personallyidentifiable information for each user of the plurality; cause a uniquefirst token to be generated using a first identifier associated with thepersonally identifiable information included in the exposure data foreach user of the plurality, each unique first token being a first levelof abstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; cause a unique second token to be generated using asecond identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality, each unique second token being a second level of abstractionof the corresponding user's personally identifiable information includedin the exposure data and uniquely identifying the corresponding user;and cause each user's personally identifiable information in theexposure data to be replaced with the user's corresponding unique secondtoken, the unique second token used to identify the corresponding user'sexposure to the online ad campaign and to identify sales transactions ofthe corresponding user in offline sales transaction data in measuringeffectiveness of the online advertisement campaign using the exposureand offline sales transaction data.

In yet another aspect, a computer readable storage medium for tangiblystoring thereon computer readable instructions that when executed causeat least one processor to maintain a user data store for each of aplurality of users, each user data store storing personally identifiableinformation about a corresponding user, the corresponding user'spersonally identifiable information being associated with a uniqueidentifier that uniquely identifies the user in ad tracking data;collect the ad tracking data for each advertisement served to theplurality of users in connection with an online advertisement campaign;generate exposure data using the ad tracking data, the exposure dataincluding at least a portion of the personally identifiable informationfor each user of the plurality; cause a unique first token to begenerated using a first identifier associated with the personallyidentifiable information included in the exposure data for each user ofthe plurality, each unique first token being a first level ofabstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; cause a unique second token to be generated using asecond identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality, each unique second token being a second level of abstractionof the corresponding user's personally identifiable information includedin the exposure data and uniquely identifying the corresponding user;and cause each user's personally identifiable information in theexposure data to be replaced with the user's corresponding unique secondtoken, the unique second token used to identify the corresponding user'sexposure to the online ad campaign and to identify sales transactions ofthe corresponding user in offline sales transaction data in measuringeffectiveness of the online advertisement campaign using the exposureand offline sales transaction data.

In accordance with one or more embodiments, a system is provided thatcomprises one or more computing devices configured to providefunctionality in accordance with such embodiments. In accordance withone or more embodiments, functionality is embodied in steps of a methodperformed by at least one computing device. In accordance with one ormore embodiments, program code to implement functionality in accordancewith one or more such embodiments is embodied in, by and/or on acomputer-readable medium.

DRAWINGS

The above-mentioned features and objects of the present disclosure willbecome more apparent with reference to the following description takenin conjunction with the accompanying drawings wherein like referencenumerals denote like elements and in which:

FIG. 1 provides an example of online advertising using computing devicesin accordance with one or more embodiments of the present disclosure.

FIG. 2 provides an example of exposure data processing in accordancewith one or more embodiments of the present disclosure.

FIG. 3 provides an example of conversion data processing in accordancewith one or more embodiments of the present disclosure.

FIG. 4 provides an example of a flow in analysis of online ad campaigneffectiveness using offline conversion data in accordance with one ormore embodiments of the present disclosure.

FIG. 5 provides a flow in accordance with one or more embodiments of thepresent disclosure.

FIG. 6 illustrates some components that can be used in connection withone or more embodiments of the present disclosure.

FIG. 7 provides is a detailed block diagram illustrating one example ofan internal architecture of a computing device in accordance with one ormore embodiments of the present disclosure.

DETAILED DESCRIPTION

In general, the present disclosure includes an online advertisementmeasurement system, method and architecture. Certain embodiments of thepresent disclosure will now be discussed with reference to theaforementioned figures, wherein like reference numerals refer to likecomponents.

In accordance with one or more embodiments, effectiveness of an onlineadvertisement campaign, which comprises one or more advertisements, ismeasured using offline sales transactions data. In accordance with oneor more embodiments, effectiveness is determined using anonymized onlinead exposure data and anonymized offline sales, or conversion, data. Theanonymized exposure and conversion data are matched using tokens fromeach data set; each token corresponds to a user and is anonymized forthe user. The anonymized exposure and conversion data are aggregatedusing the tokens, and are used to identify a level of a user's exposure,e.g., whether or not the user is exposed, to the online advertisement(s)and a user's purchase history, e.g., sales transactions of the goodand/or service being advertised. Users' online ad exposure and offlinesales history can be analyzed and used, anonymously, to gauge adeffectiveness. The effectiveness measure, or measures, can be used toconfigure another ad campaign.

To maintain privacy and provide anonymity, personally identifiableinformation (PII), or information that can be used to identify a user,is anonymized and the anonymized data is used in the exposure andconversion data, in place of the PII, to represent the user. Inaccordance with one or more embodiments, an identifier that isassociated with the PII for a user is converted to a token that is usedto match the user's exposure data to the user's conversion data.

PII can comprise identification information, such as without limitation,a user identifier, or UID, name, address, age, gender, etc. PII can becollected together with online ad activity by an online data aggregator,such as for example, an online entity with a capability to create anaccount for a user, or otherwise collect user identificationinformation, and associate online ad tracking data for the user, such asand without limitation, ad views, ad clicks, etc. In accordance with oneor more embodiments, the online entity maintains the UID that uniquelyidentifies the user and associates the UID with the user's PII. Exposuredata can comprise ad tracking data, e.g., ad views, ad clicks, etc.,aggregated ad tracking data, e.g., total number of ad views accumulatedfrom individual ad views, total ad clicks determined from individual adclicks, etc. Information about the specific online ad campaign can beincluded with the exposure data as well. Prior to anonymization, theexposure data can include PII information for users whose ad trackingdata is represented in the exposure data. The PII information for usersis replaced in the exposure data with a token representing the user. Inaccordance with one or more embodiments, the token is generated using anidentifier linked to, or associated with, the user's PII, or portionthereof.

In accordance with one or more embodiments, the identifier, e.g., UID,which is associated with a user's PII, or portion thereof, correspondingto the exposure data undergoes a first level of abstraction, to generatea first token. Using the user identifier, or UID, assigned by an onlineaggregator or publisher, e.g., Yahoo!® Inc., as a non-limiting example,the UID for each user from the exposure data is anonymized, orconverted, to an anonymous UID, as a first token, using a first tokengenerator. By way of a non-limiting example, the first token generatorcan comprise an encryption module that encrypts the UID linked to, orassociated with, the PII, or portion thereof, to generate the firsttoken. In accordance with one or more embodiments, the first token isgenerated by a first trusted third party, e.g., a marketing servicesprovider such as without limitation a credit services entity or otherentity that maintains or has access to consumer information, using thefirst token generator in response to a request by the onlinepublisher/aggregator. Using the UID example to further illustrate, inaccordance with one or more embodiments, in further response to therequest the UID undergoes another conversion to generate a second tokenusing a second token generator. By way of a non-limiting example, thesecond token generator uses encryption to generate the second tokenusing and identifier linked to, or associated with, the PII, or portionthereof. In accordance with one or more embodiments, the second tokengenerator module is provided by a second trusted third party.

In accordance with one or more embodiments, an identifier associatedwith the PII is used to generate the first and second tokens. Theidentifier used by the first trusted third party to generate the firsttoken is the UID. The first trusted third party uses the second trustedthird party's encryption to generate the second token using the firsttoken as the identifier associated with the PII used to generate thesecond token. The first trusted third party maintains a mapping betweenthe first token, the second token and the PII. The first trusted thirdparty's mapping can be used to match the PII of a sales transaction inthe conversion data with the second token.

In accordance with one or more embodiments, the conversion datacomprises sales transaction data collected by an offline sales entity,e.g., a retailer with brick and mortar sales locations, from offline andonline sales transactions, such as sales transactions involving theadvertised good, service, product, product line, etc., that occur at aretailer's brick and mortar store. While embodiments may be describedwith respect to offline sales transactions, it should be apparent thatonline sales transactions can be also be used with the offline salestransactions in measuring effectiveness of the online advertisingcampaign. Each transaction includes PII identifying the user, orcustomer, involved in the sales transaction. In accordance with one ormore embodiments, an identifier linked to, or otherwise associated with,the user's PII from the transaction is converted to the second tokenusing a second token generator. The user's PII in the conversion data isreplaced by the second token in the conversion data set. As previouslydiscussed, the second token generator, which can be provided by thesecond trusted third party, uses encryption to generate the second tokenusing the identifier linked to, or associated with, the PII, or portionthereof. In accordance with one or more embodiments, the user's secondtoken generated from the identifier associated with the PII in theexposure data matches the user's second token generated from theidentifier linked to, or associated with, PII in the conversion data.

The anonymized exposure and conversion data is aggregated and used tomeasure effectiveness of an online advertisement campaign comprising oneor more online advertisements, or ads. The tokens in the anonymizedexposure and conversion data are used to correlate user data from thetwo data sets. By way of a non-limiting example, token “001”representing one user can be used to identify data in the exposure dataand/or the conversion data for the user represented by token “001”. Inaccordance with one or more embodiments, the exposure and conversiondata can be used to create sample user populations. A control populationcomprises users that have not been exposed to the online ad campaign,and the test population comprises users that have been exposed to theonline ad campaign. One or more metrics, e.g., conversion, retention,return on investment metrics, can be determined using the test and/orcontrol populations and the one or more metrics can be used to measurethe effectiveness of the online advertisement campaign.

FIG. 1 provides an example of online advertising using computing devicesin accordance with one or more embodiments of the present disclosure.

User computing device 112 can receive one or more web pages served byweb server 102 to user computing device 112 via network 110. A web pagereceived by user computing device 112 can be displayed by computingdevice 112, e.g., using a web browser executing on the user computingdevice 112. The web page can include one or more advertisements servedby ad server 106 in accordance with parameters, or specifications, ofthe ad campaign, e.g., advertisement content or creative, targetinginformation, placement and size of advertisement within web page, timeperiod(s) (e.g., start time, end time, duration of the campaign, time(s)of day, day(s) of the week, week(s) of the month, etc.), targeting datato be collected, etc. The ad campaign can include one or moreadvertisements for a given good, service, product, product line, etc. Addatabase comprises information such as the ad campaign 118 and adtracking data, and the identifier, e.g., the unique identifier used togenerate the first token, which is associated with the PII.

Ad server 106 can serve one or more advertisements for a web page to theweb server 102 for the web server 102 to serve to computing device.Alternatively, the web page can include code, e.g., hypertext markuplanguage and/or scripting language code, to request one or moreadvertisements for display with the web page at the user computingdevice 102. The request can be made to the ad server 106, oralternatively to the web server 102, to retrieve the one or moreadvertisements from ad server 106, e.g., via ad database 108. The adserver 106 retrieves the one or more advertisements in accordance withthe ad campaign 118. The one or more advertisements are served by adserver 106, or web server 102, via network 110 to computing device 112.

Tracking data, e.g., ad view, ad click (click through) data, iscollected for the one or more advertisements. The tracking data can bereceived by ad server 106, or web server 102, and collected by ad server106 in ad database 108. For each of the one or more advertisements, thetracking data can include information that identifies the advertisement,the time of an ad view and/or ad click through, size (e.g., in pixel)and/or placement of the advertisement in the web page, whether or notthe view is the first one for the user, etc., and PII information of theuser, e.g., from user database 104. The PII can comprise the user's UIDgenerated by an online entity, e.g., Yahoo!® Inc., for the userregistered with the online entity.

By way of one non-limiting example, some portion of the PII informationcan be stored in a cookie, or other data structure, at the usercomputing device 112, or otherwise provided by web server 102 from userdatabase 104. User database 104 can include the UID as well as some orall of the user's PII information maintained by one or more entities,including the online entity that registered the user. PII can includeinformation collected for the user at registration as well as otherinformation collected from the user's online activities, e.g., pageview(s), web search(es), social interaction(s), etc.

Databases 104, 108 and 116 are shown separately. It should be apparentthat any combination of databases 104, 108 and 116 is contemplatedwithout departing from the scope of embodiments of the presentdisclosure. Similarly, any combination of servers 102, 106 and 114 iscontemplated without departing from the scope of embodiments of thepresent disclosure. In accordance with one or more embodiments, the sameor different online entity can provide any of databases 104, 108 and 116and servers 102, 106 and 114.

Online data aggregation device 114 generates online ad exposure data116. Online ad exposure data 116 includes some or all of user PII. Inaccordance with one or more embodiments, device 114 accesses one or bothof user database 104 and ad database 108 to generate online exposuredata 116. In accordance with one or more such embodiments, in additionto PII, data 116 can include information from ad campaign 118, e.g., adand/or ad campaign identification information, ad tracking data, etc.The data 116 can include the data specified by the ad campaign 118. Thead tracking data can be at any level of aggregation, e.g., token level,token grouping level, ad, or line, level, ad grouping level, ad campaignlevel. By way of a non-limiting example, the ad tracking data caninclude information that identifies the number of views of an ad, orgroup of ads, by a user, group of users, all or any combination of theusers represented in the exposure data 116. It should be apparent thatthe exposure data 116 can include any ad tracking data, at any level orcombination, and that exposure data 116 can include any PII. Inaccordance with one or more embodiments, the PII in the conversionand/or exposure data can be aggregated for all or some of the users toprovide group-level demographic, geographic, etc. information.

In accordance with one or more embodiments, the exposure and PII data116 is forwarded to a first trusted third party, e.g., a marketingservices provider such as without limitation a credit services entity orother entity, which uses an identifier, or identifiers, associated withthe PII to generate first and second tokens, and creates a mapping,e.g., between two or more of the PII data, first token data and secondtoken data. It should be apparent that this functionality can beprovided by any entity, including the entity that generates the data116. Using a trusted third party, e.g., either or both of the first andsecond trusted third parties, that is separate from another entitylimits access to the encryption used to generate a token, therebyproviding additional privacy protection. By way of a non-limitingexample, a separate trusted third party can minimize the possibility ofconverting a token, e.g., by “reverse engineering” the token, to obtainthe identifier associated with the PII. Additionally, a trusted thirdparty that is a credit service can maintain a history of PII data, e.g.,historical PII data. The historical PII data can be used in reconcilingdifferent PII data for a given user in connection with exposure dataand/or conversion data. By way of a non-limiting example, the historicalPII data can be used in identifying a correlation between user recordsin the exposure data and/or conversion data by matching current PII dataassociated with one record, e.g., an exposure data record or aconversion data record, with less current PII data used in anotherrecord, e.g., an exposure data record or a conversion data record.

FIG. 2 provides an example of exposure data processing in accordancewith one or more embodiments of the present disclosure. Exposure data116, which includes PII, is used in step 202, which performs a firstlevel of abstraction using a first identifier associated with the PII togenerate a first token. Each first token is unique for a given userrelative to the first tokens of other users. As discussed herein, step202 is performed using encryption, e.g., a first identifier associatedwith some or all of the PII is input to a first token generator thatuses encryption, such as a hash function or another encryption function,to generate the first token. At step 206, a second identifier associatedwith some or all of the PII from exposure data 116, which can be thesame or different PII used in step 202, is converted to a second token.Additionally, at step 206, exposure data 212 is generated and includesthe second token. In contrast to exposure data 116, exposure data 212uses the second token as anonymous user identification instead of, or inplace of, the PII contained in exposure data 116. The first and secondtokens generated in steps 202 and 206, respectively, are input to step208 to create a mapping 210, e.g., between two or more of the PII, firsttoken and second token data. By way of one non-limiting example, themapping 210 can map the PII data, the first token generated from theidentifier associated with the PII and the second token generated fromidentifier associated with the PII. The mapping 210 can be used as alookup table. By way of a non-limiting example, in a case that themapping 210 includes the PII, instead of regenerating the first tokenand/or the second token, given the PII, the mapping 210 can be used todetermine the first token and/or the second token. Additionally, in acase that the mapping 210 includes a mapping between the first andsecond tokens, given either the first or second token, the mapping 210can be used to determine the other token rather than regenerating theother token.

In accordance with one or more embodiments, step 206 makes use ofencryption, e.g., encryption provided by a second token generator, of asecond trusted third party. In accordance with one or more suchembodiments, the first trusted third party is a trusted third partyassociated with the online data aggregation entity and the secondtrusted third party is a trusted third party of the offline dataaggregation entity. By way of a non-limiting example, the second trustedthird party can be an entity such as a data management provider or otherentity, and the offline data aggregation entity can be an entity such asa marketing analytics provider or other entity. Of course it should beapparent that embodiments of the present disclosure are not limited toany specific online entity, offline entity and/or trusted third party.Additionally, it should be apparent that the offline entity and thesecond trusted third party can be the same. As discussed above,maintaining a separation between the online, or offline, entity and thetrusted third party facilitates enhanced privacy protection.

In accordance with one or more embodiments, data from offline salestransactions is collected, e.g., by an offline entity such as a brickand mortar sales entity. Like exposure data 116, conversion data 302includes PII. Conversion data 302 can comprise one or more point-of-salesales transaction records, each corresponding to a sales transaction andcomprises personally identifiable information for the user, or customer,involved in the sales transaction. FIG. 3 provides an example ofconversion data processing in accordance with one or more embodiments ofthe present disclosure.

At step 304, the identifier linked to the PII data associated with oneor more records in conversion data 302 is converted to a second token,and the PII data is replaced by the second token generated using theidentifier linked to the PII data. The result of step 304 is conversiondata with second token 306. Step 304 can be performed on a singletransaction or multiple transactions. In accordance with one or moreembodiments, step 304 is performed using encryption, e.g., by a secondtoken generator of the second trusted third party.

In accordance with one or more embodiments, an offline aggregatoraggregates exposure data 212 and conversion data 306, matching recordsusing the second tokens. The aggregated data is used to create test andcontrol populations, and one or more metrics are computed using theexposure data 212, conversion data 306, test population and controlpopulation. The metrics can be analyzed to provide insight into theimpact of the ad campaign on online sales, and are useful in specifyinga subsequent ad campaign.

FIG. 4 provides an example of a flow in analysis of online ad campaigneffectiveness using offline conversion data in accordance with one ormore embodiments of the present disclosure. In accordance with one ormore embodiments, the flow can be performed by an offline dataaggregator. Exposure data 212 and conversion data 306 are input to step402, which aggregates the data and matches records in the data using thesecond tokens in each data set. At step 404, test and controlpopulations are identified. At step 406, one or more effectivenessmetrics are computed for the online ad campaign. By way of somenon-limiting examples, metrics that measure conversion, or completedsales, retention of existing customers, acquisition of new customers andreturn on investment, etc. can be used to gauge the effectiveness of theonline ad campaign. It should be apparent that any metrics now known orlater developed can be used with embodiments of the present disclosure.At step 408, the one or more metrics computed at step 406 can beanalyzed to determine the effectiveness of the online ad campaign,and/or to identify parameters and/or specifications for another onlinead campaign. By way of one non-limiting example, an online ad campaign'seffectiveness can be used to determine parameters of a new ad campaign,including without limitation ad budget, pricing of each advertisement inthe new ad campaign, cost per view, cost per click, etc.

In accordance with one or more embodiments, an entity, or entities, thatcollects the exposure data causes the exposure data to be processed toremove the PII from the exposure data and replaced with the secondtoken, and causes the first and second tokens to be generated using anidentifier, or identifiers, associated with the PII. In accordance withone or more such embodiments, the exposure data collection entity, orentities, can send one or more requests to cause the first trusted thirdparty to process the exposure data with PII 116, generate the tokens,create the mapping 210, and provide the exposure data 212, whichincludes the second token in place of the PII, to the offlineaggregator.

FIG. 5 provides a flow in accordance with one or more embodiments of thepresent disclosure. At step 502, an entity that collects the exposuredata, the online publisher in this example, sends the exposure data in arequest to the first trusted third party, and causes a first trustedthird party to perform steps 504, 506, 508, 510 in response. At step504, the first trusted third party uses an identifier associated withsome or all of the PII to generate the first token using the firsttrusted third party's encryption module. In further response to therequest, the first trusted third party uses an identifier associatedwith some or all of the PII to generate the second token using a secondtrusted third party's encryption module, at step 506. In accordance withone or more embodiments, in response to the request, at step 508, thefirst trusted third party can create a mapping 210. The first trustedthird party replaces the PII with the second token in the exposure data212, and forwards the exposure data 212 to a data aggregator, at step510.

At step 512, one or more retailers collect offline sales transactiondata with PII, and send a request to a second trusted third party togenerate a second token using an identifier associated with some or allof the PII from the sales transaction data, replace the PII with thesecond token, and forward the offline sales transaction data to the dataaggregator, at step 514. The mapping between first and second tokens andPII can be used to match the PII in the conversion data with the PII inthe mapping to identify the second token.

In response to receiving the exposure data of the online entity, and theconversion data of the offline entity, at step 516, the data aggregatoranalyzes the received data to determine the effectiveness of the onlineadvertising on the offline sales by matching the second token of a givenuser contained in the exposure and conversion data. At step 518, thedata aggregator creates test and control populations, and at step 520the data aggregator uses one or more metrics to measure effectiveness ofthe online advertising, e.g., the data aggregator uses a plurality ofmetrics and measures effectiveness across the plurality of metrics.

FIG. 6 illustrates some components that can be used in connection withone or more embodiments of the present disclosure. In accordance withone or more embodiments of the present disclosure, one or more computingdevices, e.g., one or more server computing devices 602 and clientcomputing devices 604 are configured to comprise functionality describedherein. By way of some non-limiting examples, server computing device602 can be web server 102, ad server 106, online data aggregationcomputing device 114, the first trusted third party's computing device,and/or the second trusted third party's computing device. By way of somefurther non-limiting examples, client computing device 604 can be usercomputing device 112, online data aggregation computing device 114, acomputing device of the first trusted third party's computing device, acomputing device of the second trusted third party, etc. In accordancewith one or more embodiments, a server computing device 602 can act as aclient computing device 604. By way of one non-limiting example, thefirst trusted third party's server computing device can act as a clientcomputing device 604 in requesting a second token from the secondtrusted third party's server computing device.

By way of a non-limiting example, as discussed herein, web server 102and/or ad server 106, as computing device 602, can serve content to usercomputing device 112, as client computing device 604, using a browserapplication via a network, such as network 110 and/or network 606. Aserver computing device 602 is coupled to one or more data stores 608.By way of some non-limiting examples, data store 608 can be userdatabase(s) 104, ad database(s) 108, online ad exposure data with PII116, mapping 210, exposure data with second token 212, conversion datawith PII 302, conversion data with second token 306, as well as otherdata, including program code to configure server 602 to execute thefunctionality discussed herein in accordance with one or moreembodiments.

The client computing device 604 can be any computing device, includingwithout limitation a personal computer, personal digital assistant(PDA), wireless device, cell phone, internet appliance, media player,home theater system, and media center, or the like.

For the purposes of this disclosure, a computing device includes atleast one processor for executing and memory for storing program code.The computing device may be provided with an operating system thatallows the execution of software applications in order to manipulatedata. Additionally, computing device such as server 602 and clientcomputing device 604 can include a removable media reader, networkinterface, display and interface, and one or more input devices, e.g.,keyboard, keypad, mouse, etc. and input device interface, for example.One skilled in the art will recognize that server 602 and clientcomputing device 604 may be configured in many different ways andimplemented using many different combinations of hardware, software, orfirmware.

In accordance with one or more embodiments, a server computing device602 can make a user interface available to a client computing device 604via the network 606. The user interface made available to the clientcomputing device 604 can include one or more pages of content, thecontent can include one or more advertisements of an ad campaign. Inaccordance with one or more embodiments, computing device 602 makes auser interface available to a client computing device 604 bycommunicating a definition of the user interface to the client computingdevice 604 via the network 606. The user interface definition can bespecified using any of a number of languages, including withoutlimitation a markup language such as Hypertext Markup Language, scripts,applets and the like. The user interface definition can be processed byan application executing on the client computing device 604, such as abrowser application, to output the user interface on a display coupled,e.g., a display directly or indirectly connected, to the clientcomputing device 604.

In at least one embodiment, the network 606, or network 110, may be theInternet, an intranet (a private version of the Internet), or any othertype of network. An intranet is a computer network allowing datatransfer between computing devices on the network. Such a network maycomprise personal computers, mainframes, servers, network-enabled harddrives, and any other computing device capable of connecting to othercomputing devices via an intranet. An intranet uses the same Internetprotocol suit as the Internet. Two of the most important elements in thesuit are the transmission control protocol (TCP) and the Internetprotocol (IP).

It should be apparent that embodiments of the present disclosure can beimplemented in a client-server environment such as that shown in FIG. 6.Alternatively, embodiments of the present disclosure can be implementedother environments, e.g., a peer-to-peer environment as one non-limitingexample.

FIG. 7 is a detailed block diagram illustrating an internal architectureof a computing device, e.g., a computing device such as server 702 oruser computer 704, in accordance with one or more embodiments of thepresent disclosure. As shown in FIG. 7, internal architecture 700includes one or more processing units, processors, or processing cores,(also referred to herein as CPUs) 712, which interface with at least onecomputer bus 702. Also interfacing with computer bus 702 arecomputer-readable medium, or media, 706, network interface 714, memory704, e.g., random access memory (RAM), run-time transient memory, readonly memory (ROM), etc., media disk drive interface 708 as an interfacefor a drive that can read and/or write to media including removablemedia such as floppy, CD-ROM, DVD, etc. media, display interface 710 asinterface for a monitor or other display device, keyboard interface 716as interface for a keyboard, pointing device interface 718 as aninterface for a mouse or other pointing device, and miscellaneous otherinterfaces not shown individually, such as parallel and serial portinterfaces, a universal serial bus (USB) interface, and the like.

Memory 704 interfaces with computer bus 702 so as to provide informationstored in memory 704 to CPU 712 during execution of software programssuch as an operating system, application programs, device drivers, andsoftware modules that comprise program code, and/or computer-executableprocess steps, incorporating functionality described herein, e.g., oneor more of process flows described herein. CPU 712 first loadscomputer-executable process steps from storage, e.g., memory 704,computer-readable storage medium/media 706, removable media drive,and/or other storage device. CPU 712 can then execute the stored processsteps in order to execute the loaded computer-executable process steps.Stored data, e.g., data stored by a storage device, can be accessed byCPU 712 during the execution of computer-executable process steps.

Persistent storage, e.g., medium/media 706, can be used to store anoperating system and one or more application programs. Persistentstorage can also be used to store device drivers, such as one or more ofa digital camera driver, monitor driver, printer driver, scanner driver,or other device drivers, web pages, content files, playlists and otherfiles. Persistent storage can further include program modules and datafiles used to implement one or more embodiments of the presentdisclosure, e.g., listing selection module(s), targeting informationcollection module(s), and listing notification module(s), thefunctionality and use of which in the implementation of the presentdisclosure are discussed in detail herein.

For the purposes of this disclosure a computer readable medium storescomputer data, which data can include computer program code that isexecutable by a computer, in machine readable form. By way of example,and not limitation, a computer readable medium may comprise computerreadable storage media, for tangible or fixed storage of data, orcommunication media for transient interpretation of code-containingsignals. Computer readable storage media, as used herein, refers tophysical or tangible storage (as opposed to signals) and includeswithout limitation volatile and non-volatile, removable andnon-removable media implemented in any method or technology for thetangible storage of information such as computer-readable instructions,data structures, program modules or other data. Computer readablestorage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other solid state memory technology, CD-ROM, DVD, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other physical ormaterial medium which can be used to tangibly store the desiredinformation or data or instructions and which can be accessed by acomputer or processor.

Those skilled in the art will recognize that the methods and systems ofthe present disclosure may be implemented in many manners and as suchare not to be limited by the foregoing exemplary embodiments andexamples. In other words, functional elements being performed by singleor multiple components, in various combinations of hardware and softwareor firmware, and individual functions, may be distributed among softwareapplications at either the client or server or both. In this regard, anynumber of the features of the different embodiments described herein maybe combined into single or multiple embodiments, and alternateembodiments having fewer than, or more than, all of the featuresdescribed herein are possible. Functionality may also be, in whole or inpart, distributed among multiple components, in manners now known or tobecome known. Thus, myriad software/hardware/firmware combinations arepossible in achieving the functions, features, interfaces andpreferences described herein. Moreover, the scope of the presentdisclosure covers conventionally known manners for carrying out thedescribed features and functions and interfaces, as well as thosevariations and modifications that may be made to the hardware orsoftware or firmware components described herein as would be understoodby those skilled in the art now and hereafter.

While the system and method have been described in terms of one or moreembodiments, it is to be understood that the disclosure need not belimited to the disclosed embodiments. It is intended to cover variousmodifications and similar arrangements included within the spirit andscope of the claims, the scope of which should be accorded the broadestinterpretation so as to encompass all such modifications and similarstructures. The present disclosure includes any and all embodiments ofthe following claims.

1. A method comprising: maintaining, by at least one computing device, auser data store for each of a plurality of users, each user data storestoring personally identifiable information about a corresponding user,the corresponding user's personally identifiable information beingassociated with a unique identifier that uniquely identifies the user inad tracking data; collecting, by the at least one computing device, thead tracking data for each advertisement served to the plurality of usersin connection with an online advertisement campaign; generating, by theat least one computing device, exposure data using the ad tracking data,the exposure data including at least a portion of the personallyidentifiable information for each user of the plurality; causing, by theat least one computing device, a unique first token to be generatedusing a first identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality, each unique first token being a first level of abstraction ofthe corresponding user's personally identifiable information included inthe exposure data and uniquely identifying the corresponding user;causing, by the at least one computing device, a unique second token tobe generated using a second identifier associated with the personallyidentifiable information included in the exposure data for each user ofthe plurality, each unique second token being a second level ofabstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; and causing, by the at least one computing device,each user's personally identifiable information in the exposure data tobe replaced with the user's corresponding unique second token, theunique second token used to identify the corresponding user's exposureto the online ad campaign and to identify sales transactions of thecorresponding user in offline sales transaction data in measuringeffectiveness of the online advertisement campaign using the exposureand offline sales transaction data.
 2. The method of claim 1, causing aunique first token to be generated further comprising: making a request,by the at least one computing device to a trusted third party, togenerate the unique first token using the first identifier associatedwith the personally identifiable information included in the exposuredata for each user of the plurality.
 3. The method of claim 1, causing aunique second token to be generated further comprising: making arequest, by the at least one computing device to a trusted third party,to generate the unique second token using the second identifierassociated with the personally identifiable information included in theexposure data for each user of the plurality.
 4. The method of claim 1,wherein the second identifier is the unique first token.
 5. The methodof claim 1, wherein a first trusted third party is used to generate theunique first token and a second trusted third party different from thefirst trusted third party is used to generate the unique second token.6. The method of claim 1, further comprising: causing, by the at leastone computing device, creation of a mapping between each user's uniquefirst and second tokens.
 7. The method of claim 1, further comprising:causing, by the at least one computing device, creation of a mappingbetween each user's personally identifiable information, and the user'sunique first and second tokens.
 8. A system comprising: at least oneprocessor and memory, the memory storing instructions that when executedby the at least one processor cause the at least one processor to:maintain a user data store for each of a plurality of users, each userdata store storing personally identifiable information about acorresponding user, the corresponding user's personally identifiableinformation being associated with a unique identifier that uniquelyidentifies the user in ad tracking data; collect the ad tracking datafor each advertisement served to the plurality of users in connectionwith an online advertisement campaign; generate exposure data using thead tracking data, the exposure data including at least a portion of thepersonally identifiable information for each user of the plurality;cause a unique first token to be generated using a first identifierassociated with the personally identifiable information included in theexposure data for each user of the plurality, each unique first tokenbeing a first level of abstraction of the corresponding user'spersonally identifiable information included in the exposure data anduniquely identifying the corresponding user; cause a unique second tokento be generated using a second identifier associated with the personallyidentifiable information included in the exposure data for each user ofthe plurality, each unique second token being a second level ofabstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; and cause each user's personally identifiableinformation in the exposure data to be replaced with the user'scorresponding unique second token, the unique second token used toidentify the corresponding user's exposure to the online ad campaign andto identify sales transactions of the corresponding user in offlinesales transaction data in measuring effectiveness of the onlineadvertisement campaign using the exposure and offline sales transactiondata.
 9. The system of claim 8, the instructions to cause a unique firsttoken to be generated further comprising instructions to: make arequest, to a trusted third party, to generate the unique first tokenusing the first identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality.
 10. The system of claim 8, the instructions to cause a uniquesecond token to be generated further comprising instructions to: make arequest, to a trusted third party, to generate the unique second tokenusing the second identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality.
 11. The system of claim 8, wherein the second identifier isthe unique first token.
 12. The system of claim 8, wherein a firsttrusted third party is used to generate the unique first token and asecond trusted third party different from the first trusted third partyis used to generate the unique second token.
 13. The system of claim 8,the instructions further comprising instructions to: cause creation of amapping between each user's personally identifiable information, and theuser's unique first and second tokens.
 14. A computer readable storagemedium for tangibly storing thereon computer readable instructions thatwhen executed cause at least one processor to: maintain a user datastore for each of a plurality of users, each user data store storingpersonally identifiable information about a corresponding user, thecorresponding user's personally identifiable information beingassociated with a unique identifier that uniquely identifies the user inad tracking data; collect the ad tracking data for each advertisementserved to the plurality of users in connection with an onlineadvertisement campaign; generate exposure data using the ad trackingdata, the exposure data including at least a portion of the personallyidentifiable information for each user of the plurality; cause a uniquefirst token to be generated using a first identifier associated with thepersonally identifiable information included in the exposure data foreach user of the plurality, each unique first token being a first levelof abstraction of the corresponding user's personally identifiableinformation included in the exposure data and uniquely identifying thecorresponding user; cause a unique second token to be generated using asecond identifier associated with the personally identifiableinformation included in the exposure data for each user of theplurality, each unique second token being a second level of abstractionof the corresponding user's personally identifiable information includedin the exposure data and uniquely identifying the corresponding user;and cause each user's personally identifiable information in theexposure data to be replaced with the user's corresponding unique secondtoken, the unique second token used to identify the corresponding user'sexposure to the online ad campaign and to identify sales transactions ofthe corresponding user in offline sales transaction data in measuringeffectiveness of the online advertisement campaign using the exposureand offline sales transaction data.
 15. The medium of claim 14, theinstructions to cause a unique first token to be generated furthercomprising instructions to: make a request, to a trusted third party, togenerate the unique first token using the first identifier associatedwith the personally identifiable information included in the exposuredata for each user of the plurality.
 16. The medium of claim 14, theinstructions to cause a unique second token to be generated furthercomprising instructions to: make a request, to a trusted third party, togenerate the unique second token using the second identifier associatedwith the personally identifiable information included in the exposuredata for each user of the plurality.
 17. The medium of claim 14, whereinthe second identifier is the unique first token.
 18. The medium of claim14, wherein a first trusted third party is used to generate the uniquefirst token and a second trusted third party different from the firsttrusted third party is used to generate the unique second token.
 19. Themedium of claim 14, the instructions further comprising instructions to:cause creation of a mapping between each user's personally identifiableinformation, and the user's unique first and second tokens.